
(12) INTERNATIONAL ATWiCATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 




(19) World Intellectual Property Organization 
International Bureau 

(43) International Publication Date 
II April 2002 (11.04.2002) 




PCT 



II 




(10) International Publication Number 

WO 02/29715 A t 



(51) International Patent Classification 7 : 

G06F 9/44 



G06N 3/00, 



(21) International Application Number: PCT/SG00/00 i 66 

(22) International Filing Date: 3 October 2000 (03.10.2000) 



(25) Filing Language: 

(26) Publication Language: 



English 



Central, Singapore 680218 (SG). ANNAPOORNA, 
Nayak, Pangal [IN/SG); Block 234, #07-05 Choa Chu 
Kang Central, Singapore 680234 (SG). 

(74) Agents: JACOB, Sheena, R. et al.; Alban Tay Mahtani & 
De Silva, Rallies City Post Office, P.O. Box 0643, Singa- 
pore 91 1722 (SG). 



English (81) Designated States (national): CN, GB, FN, JP, SG, US. 



(71) Applicant (for all designated States except US): KENT 
RIDGE DIGITAL LABS |SG/SG|; 21 Heng Mui Keng 
Terrace, Singapore 1 19613 (SG). 

(72) Inventors; and 



Published: 

— with international search report 

For two-letter codes and other abbreviations, refer to the "Guid- 



(75) I nven tors/A p p li ca n ts ( for US only) : S I T I RA M, Ran- ance Notes on Codes and A bbreviations " appearing at the begin- 
ganatha |IN/SG|; Block 218, #02-250 Choa Chu Kang ning of each regular issue of the PCT Gazette. 



(54) Title: A SYSTEM, METHOD AND LANGUAGE FOR PROGRAMMING BEHAVIOUR IN SYNTHETIC CREATURES 



Child. 

Noivprognmuun' 



< 

IT) 



o 




Gnpfetcal or Physical Enyirnnm 
where (he story b displayed or 
e na cted . For munpU: Alter or 



(57) Abstract: The invention consists of a mark-up 
language to program behaviour oi Synthetic Creatures. 
It can also be used for creating digital stories in 
graphical environments, virtual worlds, interactive 
toy systems, and robots. It is furhter designed specify 
spatial and temporal unfolding of life-like behaviour 
in synthetic creatures, while imitating the hchaviours, 
actions and expressions of humans and other living 
creatures. 



Best Available Copy 



4SDOCID: <WO 



022971 5A1J_> 



1 



WO 02/29715 




PCT/SGOO/00166 



5 A SYSTEM, METHOD AND LANGUAGE FOR PROGRAMMING BEHAVIOUR IN 

SYNTHETIC CREATURES 

Field of the invention 

10 

The present invention relates to a system, method and new mark-up 
language to program behaviour of, for example, graphical characters, interactive 
toys and robots, and may, if desired, be used for creating digital stories in 
graphical environments, virtual worlds, interactive toy systems, and robots. 

15 

Definitions 

Throughout this specification a reference to a synthetic creature is to be 
taken as including a graphical character, digital character, toy, interactive toy, 
20 robot, industrial robot; and all or part of a real character (human or animal) 
captured in digital image, on film, video tape, or otherwise. 

Background to the invention 

25 The field of interactive toys and entertainment robotics has been 

advancing by leaps and bounds. Toys are available now that are embedded with 
computing elements, sensors and actuators, that can interact with the 
surroundings and display interesting behaviour. "Furby" (trademark) by Tiger 
Electronics is a good example. However, these toys are limited due to their lack 

30 of programmability. "Furby" can not be reprogrammed to display what the owner 
likes or wants. 

Robotic kits are available that have a programming language of their own. 
An example is the "Lego" (trademark) "Mindstorms" (trademark) robotic kit 

35 meant for children to learn design, construction and programming in an 
entertaining manner. However, the RCX programming language that 
"Mindstorms" uses is a general purpose language, and has a relatively low-level 
to enable the programming of behaviour in the robot. For example, with RCX a 
child can instruct a motor to operate for a certain period of time so that the 

40 robot can move accordingly. To achieve more complex behaviour such as, for 
example, walking of a dog-like robot, requires a lot more knowledge, skill and 
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5 time for the programmer. 

It is desirable to have a high-level robotic language to program behaviour 
that one sees in the real world such as for example, walking, waving, winking, 
and so forth. With such a language, more complex and interesting behaviour can 
10 be easily programmed for the amusement of children, such as singing and 
dancing, acting a scene, conversation amongst a group of toys, a game, mock 
combat, and so forth. 

Further, many robot languages are tied to the specific hardware of a 
15 particular robot. As a result a program written to control the behaviour of one 
type of robot cannot be used in other robots built on different hardware 
platforms. There is a need for a high-level language to program behaviour in toys 
and robots. 

20 With rapid advances in semiconductor technology, micro-sensors and 

actuators and system-on-chip technology, more sophisticated toy mechanisms 
will be developed that were hard to achieve in industrial robots a decade ago. 

Consideration of prior art 

25 

US Patent 5805784 

The disclosure of this patent is directed at the storyline of a dynamically 
generated entertainment program, such as a video game, which is generated 
using a matrix of reusable storyline fragments called substories. A set of 

30 characters that participate in the storyline established and a set of reusable 
substories is defined. Each substory represents a "fragment of a story", usually 
involving an action by a subject, where the subject is one of the characters. Most 
substories can be reused multiple times with different ones of the characters 
being the subject and different ones of the characters being the direct object of 

35 the substory. Each substory has a set of possible reaction substories, which are 
a subset of the defined substories. A plan list stores plan data indicating ones of 
the substories to be performed at specified times. An initial "seed story" in the 
form of an initial set of substories is stored in the plan list. The substories stored 
in the plan list are executed at times corresponding to their respective specified 

40 times. For at least a subset of the executed substories, the end user of the 
system is either shown a video image representing the executed substory or is 
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5 otherwise informed of the executed substory. In reaction to each executed 
substory, plans to perform additional substories are generated. The additional 
substories are from a set of possible reaction substories for each executed story. 
Each plan to perform an additional substory is assigned to a specified time, and 
plan data representing the plan is stored in the plan list. 

10 

The substories could be video or image sequences. The specification 
discloses how substories are related. However, this is related to stories on a 
multimedia screen, and is not applicable to toys and robots. This specification 
also does not disclose a high-level markup language for specifying a story. 

15 

WO9303453 

This discloses a system for the sequential performance of a prerecorded 
story including text, animations or video, and audio information. The system, 
preferably implemented in a personal computer, has a continuous mode, in 

20 which it performs the story linearly and unbroken; a wait mode, in which it 
performs loops of animation, or otherwise is on standby for commands from the 
user; and an interactive mode, in which the system performs animations, sound 
or other activities which are tangential to the linear story. Text is displayed on a 
screen of the system, along with graphics and/ or video. The text is pronounced 

25 by the system in the course of the sequential performance. When the computer 
is in the interactive mode the user may command it to repeat words which are in 
the text. The pronunciation of the words is the same as the pronunciation in the 
originally pronounced context. In both the continuous mode and the interactive 
mode, the pronounced words are highlighted. Certain animations are made 

30 inaccessible to the user, even in the interactive mode, until the user has 
executed certain prerequisite steps. Therefore, certain animations are 
interdependent or nested. The performance of a given animation may depend on 
whether a particular action has been carried out, or on whether another 
animation has already been performed, or on a random factor generated by the 

35 computer. 

US5966526 

This patent has the stated purpose of providing a simulation device for 
fostering a virtual creature. A virtual creature is grown while disciplining or 
40 training of the virtual creature occurs when a player conducts a corresponding 
treatment in response to a call or a request from the virtual creature on the 
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5 screen. The device is provided with mark display units displaying a plurality of 
kinds of treatments in respect of fostering the virtual creature. This is by marks 
and key switches for inputting a corresponding treatment by selecting a specific 
mark from the plurality of marks. The device is provided with a storing unit for 
storing control data in respect of fostering the virtual creature, a control unit 

10 reading corresponding control data from the storing unit when the treatment in 
respect of fostering the virtual creature is inputted by operating the key 
switches. The device conduct control processing in respect of fostering the 
virtual creature based on the read control data, and has with a display unit for 
displaying the fostered virtual creature. 

15 The objectives of the present invention and this patent are quite different. The 
present invention provides a programming means for specifying behaviour of 
virtual creatures in a high-level markup language. This patent specification 
provides a simulation device for fostering/ growing virtual creatures. 

20 W093 16776 

A real-time, interactive, motion-based, simulator entertainment system 
that employs a computer generated video game (or network of video games) that 
interacts with a motion-based, operator-controlled control station or simulator. 
The system employs a computer processor, helmet-mounted projection display, a 

25 motion-based cockpit, control yokes or joy sticks, a sound system, and 
computer-generated video games. A plurality of participants (typically two) 
interact with selective and timed video scenarios to achieve an objective. 
Interaction is achieved using the control yokes and buttons. Each simulator 
operates independently of the other, except that group of participants may play 

30 the same scenario, possibly at the same time by ganging or networking sets of 
simulators. Each motion-based simulator is designed and cosmetically enhanced 
to appear as an armed space vehicle, for example, and comprises an interactive, 
video scenario virtually displayed on a windshield screen allowing the 
participants to interact with the system to achieve the predetermined game 

35 objective. The simulator system incorporates selection devices (yokes or joy- 
sticks), display networks, and selection buttons and controls that permit 
interaction with the system by the participants in response to information from 
the scenarios presented on the display. 

40 This is a simulator environment for video /image games, and has very 

little in common with the present invention in terms of behaviour programming. 
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US5768122 

An object oriented approach to virtual motion programming utilizes 
generic motion identifiers such as "turn" and "accelerate" in combination with 
modifiers such as the start and end times, the magnitude, and the direction of 
the motion object to be simulated to construct universal generic motion 
descriptions of complex motion events. The minimal bandwidth requirements of 
this virtual motion programming approach enables use of MIDI communication 
standards for virtual reality show systems and interactive computer applications, 
including video games. A virtual motion controller receives generic motion 
15 descriptions and breaks down the descriptions into their constituent parts. The 
controller expands the generic motion identifiers into general acceleration 
profiles and processes the general acceleration profiles with the modifiers to 
generate ideal acceleration profiles that define the acceleration components for 
the generic motion description with reference to an ideal motion platform having 
20 unlimited mechanical movement. The controller is configured with artificial 
intelligence for learning the capabilities of a particular motion platform. The 
controller translates the ideal acceleration profiles into customized position 
commands for the motion platform based upon the controller's knowledge of the 
platform's capabilities. The controller is configured with an offset processor for 
25 optimizing the position of the platform for maximum simulation of an ideal 
acceleration profile in the future. Position processors configured with fuzzy logic 
convert the position commands from the controller into control signals for the 
actuator motors of the platform, and prevents the actuators from overextending. 



This patent discloses a high-level language /specification of generic 
descriptors for motion programming/ control of robots. 



US5617515 

A hand-held controller wand including three angular rate measurement 
sensors is coupled to a computational device for translating roll, pitch, and yaw 
data into translation and rotation signals or commands that effect the movement 
of an external device's control point, which may be an end effector of a robot. A 
transformation matrix is continually updated from the angular rate information 
obtained from the hand-held controller. This matrix is used to update a direction 
vector. A value may be stored in the computation device corresponding to a 
predetermined speed of movement. A button or other convenient control allows 
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5 the operator to control translational movement of the end effector or other 
controlled device in a direction defined by the orientation of the hand held 
controller at the predetermined speed, thus providing an intuitive control input 
for real-time direction and programming of movement. Because the present 
orientation of the wand is also known to the computational device, rotational 

10 movement may also be directed. Translation and orientation motion input may 
be provided simultaneously or separately, in accordance with the programming 
and/ or under control of the computational device. The computational device 
translates the computed direction vector into signals corresponding to 
commands that depend upon the device being controlled; such translation is 

15 especially suited for robot controllers, because such controllers accept simple 
movement commands that may readily be computed from the direction vectors. 
However, the controller is also useful for other applications in which motion 
information is required, including the motion of real objects and virtual reality. 

20 Again, this is limited to low level motion control of robots. It does not 

cover high-level behaviours such as, for example, facial expression, gestures, 
and so forth. 

US5392207 

25 A programming aid for troubleshooting real-time motion control programs 

controlling servo-motors employs a graphical control language in which 
functionally related groups of instructions are represented by icons displayed on 
a programming terminal and interconnected by sequence lines to indicate the 
order of the instruction's execution. The programming terminal receives the 

30 address of the instructions being executed and identifies this address to a 
displayed icon to modify the icon thus indicating the internal operation of the 
program on a real time basis with little interruption of the control process. 

This invention is a programming aid for troubleshooting motion control 
35 programs of servo motors and has very little to do with the present invention. 

US4843566 

A system for controlling a mechanical manipulator in a laboratory 
environment employs a system of taught motions, attribute operators and 
40 procedure rules to simplify the programming task for scientists or engineers 
skilled in their own fields and not robotics. 
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5 

The intention of this disclosure differs from the present invention in that 
the type and level of behaviour are different. For instance, this disclosure is 
concerned with moving robot manipulators and does not consider human-like 
behaviour such as, for example, facial expression, gestures, and so forth. 
10 Further, there is no sentence structure for programming behaviour. 

US6004021 

This is a toy system for use with a computer having a display. The system 
includes a plurality of interconnecting hardware toy pieces and a toy design 

15 system software for operating the computer. An inventory database stores an 
inventory of the toy pieces. An inventory database manager module updates the 
inventory database in response to player input. A design layout module creates 
and modifies a toy layout in response to player input using software 
representations corresponding to the inventory of the toy pieces and conforming 

20 to a plurality of design rules, and generates a screen indicating the toy layout. A 
layout database stores the toy layout created by the player using the toy design 
system. Other features include inventory management and control allowing a 
layout based on a fixed inventory, a layout completion module, a layout library 
module, a design assistance module, a simulation module and an education 

25 module. The hardware pieces may be moveable elements, such as a ball, and 
tracks, support pieces and stunt pieces, for guiding the ball. 

This is hardware for assembling generic elements to create a toy and 
bears relationship to programming behaviour. 

30 

CN1212169 

The electronic toy includes an external shell, a battery placed in the 
interior of the external shell for supplying electric power, a four-bit microcontrol 
unit for controlling the components of the toy, IR emitter for emitting IR signal to 

35 other toys, IR receiver for receiving IR signals from other toys, also includes four 
small keyboards respectively used for making commitment, canceling, selection 
of leftwards and rightwards movement of the cursor by means of keys, reset 
switches for restarting the game, LCD display for displaying these selections; 
and the specifications of a visitor from another planet, such as their age, weight 

40 and health, and so forth. It also has a buzzer for producing music and alarm 
signals. The electronic toy also contains a program for a game of culturing 
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5 visitors from another planet. 
US5724074 

A control program for a programmable toy (24) is created using a graphic 
programming system implemented on a personal computer (10). In the preferred 

10 embodiment, the programmable toy comprises a vehicle that can be commanded 
to make turns in 45° increments and to track an infrared source (220). To create 
the graphic control program, the user simply selects graphic objects from a 
control panel (154) presented on a cartoon-like graphic screen. For example, the 
user can select a horn graphic object (168) or a headlight graphic object (166) to 

15 respectively activate a horn or headlights on the programmable toy. The user 
creates a graphic control program on a grid (160) using a mouse (23) to select 
the graphic objects and indicate parameters. Once the graphic control program 
is completed, it is downloaded from the RS-232 serial port of the personal 
computer into the programmable toy through a connecting link (25), which is 

20 then uncoupled from the programmable toy. A switch (29) on the toy activates 
the control program. Also included in the system is a calibration template (200) 
calibrating the programmable toy to turn accurately on a specific surface. 

This has limited motion/ behaviour capabilities. It does not provide a 
25 language for narrating stories or behaviour. It's only similarity to the present 
invention in that it provides a visual tool for programming toys. 

Alice is a three-dimension, virtual-world environment with a simple 
graphical user interface ("GUI") for animating objects in the virtual world. The 
30 GUI provides buttons for primitive operations such as moving and rotating 
objects. In addition, it has "Python" (trademark), an object-oriented language for 
more complex animation that is not possible with the GUI alone. However, one 
must have a good knowledge of graphics and programming to be able to achieve 
decent character animation and behaviour in Alice. 



35 



40 



Jack is a sophisticated human modeling and simulation software 
environment developed in the University of Pennsylvania. The sophistication of 
Jack does not lend itself for behaviour programming and story creation, for 
children and lay persons. 
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The present invention is intended to specify spatial and temporal 
unfolding of life-like behaviour in synthetic creatures, while imitating the 
behaviour, actions and expressions of humans, and other living creatures. It is 
10 intended to be a relatively simple and relatively easily understandable language 
for children, and other people who may be uninitiated in programming, to be 
able to exercise their creativity, and obtain some enjoyment. 

The present invention may cover some or all the following behaviour 
15 categories: 



• Speech and Speech Intonation - includes words, phrases and sentences 

that the synthetic creature utters to communicate with other humans, 
other creatures, or other S3'nthetic creatures. Speech intonations are 
20 tonal variations in speech that emphasize and express the emotions of 

the synthetic creature. 



• Facial Expression - includes various facial expressions that convey 

emotions such as happiness, sadness, surprise, anger, delight, fear, love 
25 and so forth. 



• Gesture - includes various hand, head and other body part, gestures. 

• Gaze - includes movements of the eyes in order to look at an object, 
30 human, creature or synthetic creature or look away from it, and also to 

communicate with it. 



• Motion - includes movement from one location to another, approaching 

or retreating, entering or departing, crawling, walking, jogging, running, 
35 skipping, and so forth. 



40 



It is preferred that the present invention is implemented in Extensible 
Mark-up Language (XML). 

The mark up language implementation of the present invention may 
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5 include, in addition to text, graphics, audio, video and any other multimedia 
information. 

Using the present invention, or a visual programming tool based on the 
present invention, a person may create a story or sequence of actions of one or 
10 more synthetic creatures in a graphical or physical environment. This may be 
downloaded from a network such as the Internet so that people can share 
stories, scripts, or the like. 

The present invention may be a high-level language to script a story or 
15 set of behaviours. Preferably the present invention is implemented in the target 
environment, graphical or physical, using its own lower level language by a third 
party software or hardware company. 

When using the present invention, a person may write a script to specify 
20 the behaviour of a toy or any other type of synthetic creature. The software 
within the toy is expected to carry out the actions as specified by the script 
based on its own capabilities. The present invention, or an authoring tool 
developed based on the present invention, may be used to script a sequence of 
behavior of one or more graphical cartoon characters. Such a script can then be 
25 viewed elsewhere by using a browser that understands the present invention to 
carry out the graphical animation of the cartoon. 

Although the present invention is primarily directed at human-like 
behaviour, it is intended to include behaviour of other living creatures. 

30 

Description of the drawings 

In order that the present invention may be readily understood and put 
into practical effect there shall now be described by way of non-limitative 
35 example only a preferred embodiment of the present invention, the description 
being with reference to the accompanying illustrative drawing, in which: 

Figure 1 is a block diagram of the software architecture for implementing 
the present invention; and 

40 

Figure 2 is a preferred Visual Behaviour Programming Tool. 
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Description of preferred embodiment 

The present embodiment has tags or notations for narrating a story or a 
small episode of behaviour of humans and other living creatures such as, for 
example, dogs and cats. The version described here has only human behaviour 
elements. However, the present invention is intended to include behaviour of 
other living creatures. 

The present embodiment is an open, flexible, extensible language for 
15 creating and, possibly, exchanging stories for replaying in disparate mediums 
such as graphical animation, virtual worlds, interactive toys and robots. 

A story in the present invention is denoted by the tag <story>. A story 
may or may not have a title. A story is defined as one or more consecutive scenes 
20 and scripts, denoted by <scene> and <script>, respectively. Each scene must 
have a script following it. 

A scene specifies the initial arrangement of the synthetic creature or 
creatures, and other objects, for a script. A scene is specified before a 
25 corresponding script. Several corresponding scenes and scripts form a story. As 
such, a <scene> is a collection of synthetic creatures with their relative 
positions. Synthetic creatures or other objects are represented graphically by two 
dimensional or three dimensional objects, or physically by toys or robots. These 
synthetic objects are denoted by the tag <anyObject>. <anyObject> may 
30 represent a synthetic creature such as a man, woman, child, dog, cat, and so 
forth; or an inanimate object such as a table, chair, house, apple, and so forth. 
Every anyObject has attributes, name and objectID to uniquely identify the 
object. The relative position of an object with respect to another object in a scene 
is denoted by a <prepositionPhrase>. One or more prepositionPhrases can be 
35 used to accurately specify the relative position of an object. Preposition consists 
of either a <measure> or a <preposition>. Measure is the distance in certain 
units such as, for example, centimeters, meters, and so forth; in a certain 
direction such as, for example, left, right with respect another anyObject. 
Preposition on the other hand is an English preposition word, such as near, 
above, below, and so forth, to describe the location of an object relative to 
another object. 
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5 

A script consists of a collection of sentences that describe the behaviour 
or actions of synthetic creatures and other objects in sequence (one after 
another) or together (at once). Behaviours or actions that take place in sequence 
or serially are enclosed within a block denoted by the <doSerial> tag. Behaviours 

10 or actions that take place simultaneously or in parallel are enclosed within a 
block denoted by the <doParallel> tag. A <sentence> describes a single behaviour 
or action of a synthetic object. For example, "Monkey ate the apple" is a valid 
sentence. A sentence in the present invention consists of one mandatory 
<subject>, one mandatory <verb>, one optional <indirectObject>, one optional 

15 <directObject> and one or more optional <prepositionPhrase>, in that order. 

A subject is an anyObject. A verb denotes a behaviour or action of the 
subject. It should be noted that verbs of the present tense form are sufficient to 
describe stories and behaviour for the purpose of replaying them in graphical or 
20 physical environments. All past tense and future tense sentences can be re- 
framed into the present tense without loss of effect. 

Following are the major classes of verbs that can be specified: 

25 • Speech and Speech Intonation - includes verbs that denote how the 

speech is delivered in order that the synthetic creature communicates 
with other objects and/or synthetic creatures, a living creature. Speech 
intonations are tonal variations in speech that emphasize what is said, 
and express the emotions of the synthetic creature. 

30 

• Facial Expression - includes verbs that denote various facial expressions 
that convey emotions such as, for example, happiness, unhappiness, 
sadness, joy, surprise, imitation anger, delight, tenderness, fear, loathing, 
love, pain, detatched, enthusiastic, unpassioned, passion. 

35 

• Gestures - includes verbs that denote various hand and head gestures, 
and gestures of other body parts such as, for example, shoulders, arms, 
forearms, fingers, thumbs, chest, stomach, back, posterior, legs, ankles, 
feet, buttocks, eyebrows, nose, mouth. 

40 
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• Gaze - includes verbs that denote movements of the eyes in order to look 
at an object, human creature or synthetic creature, or look away from it, 
and also to communicate with another synthetic creature or living 
creature. 

• Motion - includes verbs that denote movement from one location to 
another including, for example, approaching, retreating, entering, 
arriving, leaving, departing, crawling, walking, jogging, running, skipping, 
swimming, flying, falling, talking, trembling, jumping, tumbling, 
exercising, and so forth. 

Based on the above classification, there are five types of verbs: 

<speechVerb>, <facialExpressionVerb>, <gestureVerb>, <gazeVerb> and 
<motionVerb>. Some common verbs are listed in Table 1 at the end of this 
description. The verbs listed are the most common body motion verbs that can 
be used to describe behaviour or tell a stoiy. Gesture and gaze verbs were 
chosen from the book "Encyclopaedia of World Gestures" by Don Morrison. 
Naturally, these are by way of example only and many other verbs can be 
included in the present invention . 

In a program of the present invention, a user can include text strings for 
dialogue or words spoken by the subject enclosed within the speechVerb tags 
that denote how the dialog is to be delivered. When the story is played or enacted 
in a graphical or physical environment, the words or other sounds spoken or 
otherwise created may be delivered using a Text-to- Speech- Synthesizer. 

It is to be noted that the present invention is not limited to speech in the 
English language. The present invention includes speech specification and 
synthesis in other languages including (but not limited to) Hindi, French, 
German, Spanish, Portuguese, Italian, Greek, Arabic, Chinese, Japanese, and so 
forth. The language for any particular item of speech may be user- selected. In 
order to facilitate this, the present invention may include an attribute for the 
speechVerb called "spokenLanguage". An implementation of this in XML could 
be: 
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5 <!ATTLIST speech Verb spokenLanguage (english | hindi | german | french | Chinese) 
CD ATA #IMPLIED>. 

There are a number of attributes common to all the verbs. All the verbs 
may have an adverbSpeed property, that denotes the speed with which the 
10 behaviour or action is to be carried out. There may be three degrees to the 
adverbSpeed: slow, normal and fast, with normal being the default value. 
However, other values may be used such as, for example, moderate, accelerate, 
and so forth. Another common attribute is the number OfTimes to denote how 
many times is the action carried out. The default may be one. 



IndirectObject and directObject are anyObjects that are based on the 
definition of indirect objects and direct objects in English grammar. According to 
correct English grammar, two kinds of objects follow verbs: direct objects and 
indirect objects. To determine if a verb has a direct object, one would isolate the 

20 verb and make it into a question by placing "whom?" or "what?" after it. The 
answer, if there is one, is the direct object. An indirect object is the recipient of 
the direct object, in a sense. To determine if a verb has an indirect object, one 
would isolate the verb and ask "to whom?", "to what?", "for whom?", or "for 
what?" after it. The answer is the indirect object. In the example below, 'gives' is 

25 the verb; 'Sheela' is the indirect object; and 'bouquet of flowers' is the direct 
object. 

Example: Mike gives Sheela a bouquet of flowers. 

30 Using the sentence structure described above, various types of sentences 

can be formed that describe actions, as illustrated below: 



15 



Subject & Verb Sentence 
Example: Child cries. 



35 



Here 'child' is the subject and 'cries' is the verb. 



Subject, Verb & DirectObject Sentence 
Example: Monkey eats a banana. 

*Monkey' is the subject, 'eats' is the verb, 'banana' is the DirectObject. 



40 



MMSDOCID <WO 



WO 02/29715 




PCT/SG00/00166 



15 

Subject, Verb, DirectObject & IndirectObject Sentence 
Example: Mike gives Alice a bouquet of flowers. 

Mike' is the subject, 'gives' is the verb, 'Alice 1 is the IndirectObject', and 'a 
bouquet of flowers' is the DirectObject. 

Subject, Verb, DirectObject, IndirectObject & PrepositionPhrase Sentence 
Example: Mike gives Alice a bouquet of flowers near the window. 
'Mike' is the subject, 'gives' is the verb, 'Alice' is the IndirectObject'. 'a 
bouquet of flowers' is the DirectObject, and 'near the window' is the 
prepositionPhrase. In the prepositionPhrase, 'near' is the preposition and 
Svindow' is anyObject. 

Subject, Verb & PrepositionPhrase Sentence 
Example: Mike eats inside the kitchen. 

Ivlike' is the subject, 'eats' is the verb, and 'inside the kitchen' is the 
prepositionPhrase . 

It is to be noted that the present invention is not limited to the use of 
English language in specifying the story or behaviour. The present invention 
could be used to describe a story or specify behaviour in other languages of the 
25 world, such as those described earlier. The language may be user- selected. In 
order to facilitate this, in one embodiment of the invention, the <story> tag could 
have an attribute called "stoxyLanguage" with one language to be selected from 
the attribute list. For example, the XML definition for this could be as follows: 

30 <!ATTLIST story story Language (english | hindi | french | german | Chinese) CD ATA 
#IMPLIED> 

The structure of a sentence in the script then depends on the language 
selected. Different languages of the world may have different sentence structure 
35 and grammar. The present invention could be implemented with language 
structures for different languages. Although, in the present embodiment given, 
only English language implementation is shown, the invention is not limited to 
English language as such. 

40 It is preferred that the present invention is implemented in XML. An 
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5 example of the Document Type Definition (DTD) is given in Table 2. 

To illustrate how the present invention can be used to program a story, 
Table 3 is a brief story, and Table 4 is a program of the folktale in accordance 
with the present invention. 

10 

Figure 1 shows the block diagram of the software architecture of the 
present invention. 

Although a simple, high-level markup language, it may be even simpler 
15 for children and non-programmers to use a visual tool for programming. Such a 
visual tool can provide buttons, icons and menu items for creating a story. The 
visual tool can then generate the script as output. Programmers, on the other 
hand, can use their favourite text editor to create the program. 

20 The program so created will then be validated and parsed by an XML 

parser. There are many commercially, as well as freely available XML parsers. 
For example, MSXML from "Microsoft" (trademark). 

If the program is valid as determined by the parser, no error messages 
25 are generated, and the parser generates a tree of objects. If the program is not 
valid, the parser generates relevant error messages that will need to be corrected 
in the visual tool or the text editor. 

The next step is the enactment of the story in the target environment, 
30 whether graphical or physical. For example, an Alice-Python Code Generator 
could be used that would take the tree of objects as input, and generate Python 
code for execution under the Alice graphical virtual world. As an illustration, 
Table 5 shows sample Python code for Alice, for a human walking routine. This 
routine will be called when the "walk" verb is used in a story. 

35 

Similarly, a Lego Mindstorms code generator could be used that would 
generate RCX code or NQC code or LegOS code for excution in the Lego 
Programmable Bricks. 

40 The architecture is extensible so that future virtual worlds or toy 

environments can be implemented. 
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5 

A visual behaviour programming tool ("VBPT") can be used to create 
stories visually. The VBPT can generate the code when a story is created. The 
code can then be fed to the XML parser for further processing as described 
above. Such a tool is shown in Figure 2. It has a menu bar 10; scene 
10 preparation window 12; script creation window 14; program window 16; and a 
story display window 18. 

The menu bar 10 is used to start a story with a title; create multiple 
scenes and scripts; number them in order so as to sequence them; save a story; 
15 open an existing story; and so forth. The menu bar 10 also has menu items for 
choosing functions available as icons and buttons in other windows described 
below. 

As the name suggests the scene preparation window 12 provides tools for 
20 preparing a scene before programming the script in the story. The window 12 
has buttons and icons 20 to add synthetic objects into a scene, and to move and 
rotate the objects around in order to position them where necessary. The story 
display window 18 can display the objects in the scene as they are moved in 
order to provide visual feedback to the user. This is similar to the user interface 
25 in the Alice Virtual World and other visual tools. 

The script preparation window 14 and program window 16 are used 
hand-in-hand for programming behaviour of synthetic creatures. The script 
preparation window 14 has four main boxes 22, 24, 26 and 28 arranged from top 

30 to bottom on the right hand side of the VBPT: program control box 22, objects 
box 24, verbs box 26, prepositions box 28; as well as speech text input window 
30. Buttons and icons 32 are provided in the boxes (except in the speech text 
input 30 box) to enable a user to click on an icon or button and choose an 
appropriate program item. Each tool box 24, 26, 28 has arrows 34 to browse the 

35 items available, as there may be more items than the tool box 24, 26, 28 can 
display at once. 

The topmost tool box is a program control box 22, and has rectangular 
buttons 36, 38, 40 respectively named doSerial 36, doParallel 38 and sentence 
40 40. When any of these buttons 36, 38, 40 are clicked by the user, corresponding 
program items appear in the program window 16. For example, if user presses 
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5 the doSerial button 36, a doSerial rectangle 42 appears in the program window 
16 within which several sentences can be contained. 

When the sentence button 40 is clicked, a sentence template 44 appears 
in the program window 16. A sentence consists of empty boxes 46 representing 
10 subject, verb, indirectObject, directObject, preposition and object tuples, each 
tuple representing a prepositionPhrase. To fill up the template, the user would 
have to click on one of these sentence elements 46 and click on a corresponding 
icon or button in the objects tool box 24, verbs tool box 26, or preposition tool 
box 28. 

15 

Whenever the user chooses a speech related verb from the verbs tool box 
26, the user will have to key in the dialogue or words to be uttered in the speech 
text box 30 at the lowest right-hand corner of VBPT. 

20 In addition, windows may be displayed to request input depending on the 

type of sentence element (object, verb or preposition) chosen. For example, if a 
verb is chosen, a window would be displayed requesting input for adverbSpeed 
and numberOfTimes according to the description of verbs above. 

25 The story display window 18 window displays the story (both the scenes 

and script) as it is being programmed using the VBPT. In this way, the user has 
immediate feedback on the story being created. Immediate feedback on what is 
being programmed is preferable for children and other non-programmers. 

30 As the present invention is, in one form, a general behaviour 

programming language, applicable to various target environments, whether 
graphical or physical, the story display window 18 may be varied or adapted for 
the target environment. For example, VBPTs for "Alice" and "Lego" "Mind storms" 
may be different to that illustrated. 

35 

Authoring tools based on the present invention may be used to program 
behavior. Special purpose authoring tools for different purposes such as 
programming behaviour in toys, and for creating graphical animations, may be 
used. 

40 

With the present invention, the interactive toys landscape may change. 



02297 15A1_I_> 



WO 02/29715 




PCT/SG00/001 66 



19 

5 Toy makers will be able to create and manufacture toys with advanced 
capabilities such as mobility, expression, sensing, speech, vision, and so on. 
They may be able to write the software layer that can understand the 
instructions of the present invention and carry-out relevant actions. Creative 
content providers may be individuals or companies that use the present 

10 invention to create interesting scenarios including toys capable of activities such 
as dancing and singing; toys which can play games, plays and skits, and carry- 
on conversation, role playing, and so on. For example, popular stories or 
cartoons can be directly adopted to create puppet shows. Content providers may 
host the scripts on their websites and toy consumers may download them into 

15 their toys to display the behavior. 



Whilst there has been described in the present invention a preferred 
embodiment of the present invention, it will be understood by those skilled in the 
technology that many variation or modifications in details of design or 
construction may be made without departing from the present invention. 
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5 



Type of Verb 


Sample set of verbs 


speechVerb 


Say, recite, sing, scream, shout, mutter, and stutter. 


motionVerb 


Walk, sit, stand, run, lie down, skip, dance, throw, catch, 
count, look, eat, drink, drive, ride, swim, read, think, hold, lift, 
pull, push, drop, touch, caress, beat, pat, strike, move, roll, 
turn, wipe, write, pickup, putdown, place, climbup, 
climbdown, jump, shake, scratch, rub, stab, shrug, bite, and 
cover. 


GestureVerb & 
gazeVerb 


ArmsRaise, armsGrasp, armsAkimbo, armsFold, bellyPat, 
bellyRub, cheekKiss, cheekSlap, chestPoint, chestBeat, 
earCup, earThumb, earBlock, earCover, eyeWink, eyeWipe, 
eyeBrowFlash, eyesClose, eyesBlink, eyesStare, eyesRaise, 
eyesWeep, faceCover, fingersClasp, fingersUnclasp, 
fingerOpen, fingersClose, fingersPoint, fingersWave, fistBeat, 
fistClench, foreFingerBeckon, foreFingerBlow, foreFingerPoint, 
foreFingerRaise, foreFingerWag, foreFingerBang, hairClasp, 
handBeckon, handSalute, handshake, handWag, handWave, 
hatRaise, headNod, headShake, palmHighSlap, palmSlap, 
palmContact, thumbDown, thumbUp, thumbSuck, and 
waistBow. 


facialExpression 
Verb 


ShowNeutral, showHappiness, showSadness, showSurprise, 
showAnger, showFear, showDespair, and 
showDisappointment. 



TABLE 1 

10 
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5 



TABLE 2 



<?xml version = "1.0" standalone= u yes"?> 
<!DOCTYPE story [ 
JO <!ELEMENT story (scene, script)+> 

<! ATT LI ST story title CD ATA #IMPLIED> 

<! ELEMENT scene ( (any Obj ect | def Object), prepositionPhrase*)*> 

<! ELEMENT any Obj ect EMPTY> 

<!ATTLIST anyObject objectld ID #REQUIRED> 
15 <!ATTLIST anyObject objName CDATA # REQUIRED > 

< [ELEMENT prepositionPhrase (location, (anyObject | defObject))> 

<!ELEMENT defObject EMPTY> 

<!ATTLIST defObject ref IDREF #REQUIRED> 

<!ELEMENT location (measure | preposition) > 
20 <!ELEMENT measure EMPTY> 

<!ATTLIST measure direction (left | right | top | bottom | front | back) #REQUIRED> 

<!ATTLIST measure integerDistance CDATA "1"> 

<!ATTLIST measure unit (CM | MT | KM) "KM" > 

<!ELEMENT preposition EMPTY> 
25 <! ATT LI ST preposition word 

(near (from [above | across | between |on (under | through |next_to |of |at 

(behind (about (around (over (besides (into |by (beyond [below | inside) 

#REQUIRED> 

<!ELEMENT script (doSerial | doParallel)+> 
30 <!ELEME.\T doSerial (sentence)+> 

<!ELEMENT doParallel (sentence)+> 

<! ELEMENT sentence (subject, verb, indirectObject?, 

directObject?, prepositionPhrase*) > 

<!ELEMENT subject (anyObject | defObject) > 
35 <! ELEMENT verb (speechVerb | motionVerb | gestureGazeVerb | 

facialExpressionVerb)> 

<!ATTLIST verb adverbSpeed (slow) normal) fast) "normaT> 
<IATTLIST verb numberOfTimes CDATA "0"> 
< I ELEMENT speechVerb (#PCDATA)> 
40 <!ATTLIST speechVerb verb 

(say | recite | sing | scream | shout | mutter | stutter | exclaim) "say"> 
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5 <! ELEMENT motionVerb EMPTY> 
<!ATTLIST motionVerb motion 

(walk | sit | standj run| lie_down| skip) dance | throw) catch | count | look) 
eat | drink) drive | ride| swim) read | think) hold] lift | pull) push | drop) 
touch) caress) beat) pat| strike) move| roll| turn| wipe| write) pickup) 
10 putdown) place) climbup) climbdown) jump| shake | scratch) rub) stab | 
shrug | bite | cover | open | give | take) #REQUIRED> 
<! ELEMENT gestureGazeVerb EMPTY> 

<!ATTLIST gestureGazeVerb gestureGaze (ArmsRaise | armsGrasp| armsAkimbo| 

armsFold) bellyPat| bellyRub) cheekKiss| cheekSlap) chestPointj chestBeat) 
15 earCup) earThumb| earBlock| earCover| eye Wink) eye Wipe | eyeBrowFlash | 

eyesClose) eyesBlink | eyesStare) eyesRaise| eyesWeep | faceCover) 

fingersClasp | fingersUnclasp | fingerOpen | fmgersClose | fingersPoint | 

fingers Wave | fistBeat ) fistClench | foreFingerBeckon | foreFingerBlow | 

foreFingerPoint | foreFingerRaise | foreFingerWag | foreFingerBang | hairClasp | 
20 handBeckon) handSalute| handShake| handWag) handWave| hatRaise| 

headNod) headShake) palmHighSlap | palmSlap| palm Contact) thumbDown| 

thumbUp | thumbSuck | waistBow) #REQUIRED> 

<! ELEMENT facialExpressionVerb EMPTY> 

<!ATTLIST facialExpressionVerb facialExpression 
25 (showNeutral | showHappiness | showSadness | showSurprise 

| showFear | show Anger | showDespair) " showNeutral" > 

<! ELEMENT indirectObject (anyObject | defObject)> 

<! ELEMENT directObject (anyObject | defObject)> 

] > 

30 TABLE 2 (CONCLUSION) 
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The Purse of Gold 

10 

A beggar found a leather purse that someone had dropped in the 
marketplace. Opening it, he discovered that it contained 100 pieces of gold. Then 
he heard a merchant shout, "A reward! A reward to the one who finds my leather 
purse!" 

15 Being an honest man, the beggar came forward and handed the purse to 

the merchant saying, "Here is your purse. May I have the reward now?" 

"Reward?" scoffed the merchant, greedily counting his gold. "Why the 

purse I dropped had 200 pieces of gold in it. You've already stolen more than the 

reward! Go away or I'll tell the police." 
20 "I'm an honest man," said the beggar defiantly. "Let us take this matter to 

the court." 

In court the judge patiently listened to both sides of the story and said, "I 
believe you both. Justice is possible! Merchant, you stated that the purse you 
lost contained 200 pieces of gold. Well, that's a considerable cost. But, the purse 
25 this beggar found had only 100 pieces of gold. Therefore, it couldn't be the one 
you lost." 

And, with that, the judge gave the purse and all the gold to the beggar. 



30 
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TABLE 4 

<stoiy title='The Purse Of Gold"> 
<scene> 

10 <anyObject objectId="ol" objName^'sky" /> 

<anyObject objectld- 'o2" objName="beggar" /> 
<prepositionPhrase> 
<location> 

<measure direction-'bottom" integer Distance-' 10" /> 
15 </ location > 

<defObject ref="ol M /> 

< / prepositionPhrase> 

<anyObject objectId= ,, o3 ,, obj Name-' merchant" /> 

< prepo sitionPhr ase > 
20 <location> 

<measure direction="left" integerDistance="2" /> 
</location> 
<defObject ref= M o2 M /> 

< / prepositionPhrase> 

25 

< / scene> 
<script> 

<doSerial> 

< sentence > 
30 < subject > 

<anyObject objectId="o4 1 ' objName-'purse" /> 
</subject> 

<verb> 

35 <motionVerb motion= u drop" /> 

</verb> 

<prepositionPhrase> 
<location> 

<preposition word="on M /> 
40 </location> 

<anyObject objectId="o5" objName=" ground" /> 
</prepositionPhrase> 
<prepositionPhrase> 
<location> 

45 <preposition word-'from" /> 

</location> 

<anyObject objectId="o6" objName-' pocket" /> 

< / prepo sitionPhrase> 

50 </sentence> 
< sentence > 
< subject > 

<defObject ref= ,, o2" /> 
</ subject > 



55 



<verb> 

<motionVerb motion= ,, walk" /> 
</verb> 
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5 <prepositionPhrase> 

<location> 

<measure direction^" front" integerDistance- '6" unit-'CM" /> 
< /location> 
<defObject ref="o4" /> 
10 < / prepositionPhrase> 

</ sentence > 

<sentence> 
15 < subject > 

<defObject ref="o2 M /> 
</ subject > 
<verb> 

<motionVerb motion- 1 look." /> 
20 </verb> 

<directObject> 

<defObject ref= T, o4" /> 
</directObject> 
</sentence> 

25 

<sentence> 
< subject > 

<defObject ref="o2" /> 
</subject> 
30 <verb> 

<motionVerb motion= 1, pickup" /> 
</verb> 
<directObject > 

<defObject ref="o4" /> 
35 </directObject> 
</sentence> 

<sentence> 
< subject > 
40 <defObject ref= ,, o2 ,, /> 

</subject> 
<verb> 

<motionVerb motion-' open" /> 
</verb> 
45 <directObject> 

<defObject ref="o4" /> 
< / directObject> 
</ sentence > 

50 <sentence> 

< subject > 

<defObject ref="o2 n /> 
</ subject > 
<verb> 
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5 <speechVerb verb-' exclaim" > 

"This purse contains 100 pieces of gold !" 
< / speechVerb> 
< / verb> 

10 </sentence> 

<sentence> 
< subject > 

<defObject ref="o3 h /> 
15 </subject> 

<verb> 

<speechVerb verb-' shout" > 

"A reward! A reward to the one who finds my leather 
purse!" 

20 < / speechVerb> 

</verb> 
</ sentence > 

<sentence> 
25 < subject > 

<defObject ref="o2" /> 
</subject> 
<verb> 

< motion Verb motion- 1 walk" /> 
30 </verb> 

<prepositionPhrase> 
<location> 

<preposition word="near" /> 
</location> 
35 <defObject ref= ,, o3" /> 

</prepositionPhrase> 
</sentence> 



</doSerial> 



<doParallel> 

<sentence> 
< subject > 

<defObject ref="o2" /> 
45 </subject> 

<verb> 

<motionVerb motion="give" /> 
</verb> 

<indirectObject > 
50 <defObject ref="o3" /> 

</indirectObject> 
<directObject > 

<defObject ref="o4" /> 
< / directObject> 



</ sentence > 
<sentence> 
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5 < subject > 

<defObject ref= M o2" /> 
</subject> 
<verb> 

<speechVerb verb-' say" > 
10 "Here is your purse. May I have the reward now?" 

< / speechVerb> 
</verb> 
</sentence> 
<sentence> 
15 < subject > 

<defObject ref="o3" /> 
</subject> 
<verb> 

<motionVerb motion-' take" /> 
20 </verb> 

<directObject > 

<defObject ref="o4" /> 
</directObject> 

25 </sentence> 

</doParallel> 
<doSerial> 

< sentence > 
30 < subject > 

<defObject ref="o3" /> 
</ subject > 
<verb> 

<motionVerb motion="open" /> 



35 



</verb> 
<directObject > 

<defObject ref="o4" /> 

< / direc tObj ec t > 
40 </ sentence > 

< sentence > 
< subject > 

<defObject ref="o3" /> 
</ subject > 
45 <verb> 

<speechVerb verb-' scream" > 

" Reward ? Why the purse I dropped had 200 pieces of gold in it. 
You've already stolen more than the reward! Go away or 111 tell the 
police." 

50 < / speechVerb> 

</verb> 

< / sentence> 
<sentence> 

55 <subject > 

<defObject ref="o2" /> 

TABLE 4 (PART) 
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5 </subject> 
<verb> 

<speechVerb verb="say" > 

" I'm an honest man, Let us take this matter to the court." 
</ speech Verb> 
10 </verb> 

</sentence> 

</doSerial> 
15 < / script> 

< scene > 

<defObject ref= n o3" /> 
<prepositionPhrase> 
20 <location> 

<preposition word="on M /> 

</location> 

<defObject ref= M o5" /> 

< / prepositionPhrase> 
25 <defObject ref^'oS" /> 

< prepo sitionPhrase> 

<location> 

<measure direction="left" /> 

30 </location> 

<defObject ref="o2" /> 

< / prepo sitionPhrase> 

<anyObject objectId= ,, o7 ,, obiName^'judge" /> 

< prepo sitionPhrase> 
35 <location> 

<measure direction="front" / > 

</location> 
<defObject ref= ,, o2" /> 
40 < / prepositionPhrase> 

</scene> 
<script> 

<doParallel> 
45 <sentence> 

< subject > 

<defObject ref= r, o2" /> 
</subject> 
<verb> 

50 <motionVerb motion^'look" /> 

</verb> 
<directObject > 

<defObject ref="o7 M /> 
55 </directObject> 
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5 < / sentence> 

<sentence> 
< subject > 

<defObject ref="o2" /> 
</ subject > 
10 <verb> 

<speechVerb verb- 'say" > 

"Sir , I found this purse on the road in the market place .It had 100 
pieces of gold . I heard this merchant shout that he would give reward 
to anyone who finds and gives back his purse. I honestly returned his 
15 purse . But he didn't give my reward ". 

< / speechVerb> 

</verb> 

</sentence> 

20 </doParallel> 
<doParallel> 
<sentence> 
< subject > 

<defObject ref="o3" /> 
25 </subject> 

<verb> 

<motionVerb motion-'look" /> 
</verb> 

30 <directObject > 

<defObject ref="o7" /> 
</directObject> 



35 



</sentence> 



< sentence > 
<subject > 

<defObject ref="o3" /> 
</subject> 
40 <verb> 

<speechVerb verb="say H > 

"The purse contained two-hundred pieces of gold. The beggar stole 
100 pieces of gold. So the beggar got his reward and there is now 
no need to reward him" 
45 </ speechVerb> 

</verb> 
</ sentence > 
</doParallel> 

50 <doSerial> 

<sentence> 
<subject > 

<defObject ref="o7" /> 
</subject> 
55 <verb> 
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02297 15A1_I_> 



WO 02/29715 




PCT/SG00/00166 



30 

5 < speech Verb verb="say"> 

"I believe you both. Justice is possible! Merchant, you stated that 
the purse you lost contained 200 pieces of gold. Well, that's a 
considerable cost. But, the purse this beggar found had only 100 
pieces of gold. Therefore, it couldn't be the one you lost." 
10 < / speechVerb> 

</verb> 
</sentence> 
<sentence> 
< subject > 
1 5 <defObject ref="o7 M / > 

</subject> 
<verb> 

<motionVerb motion =,, give" /> 
</verb> 
20 <indirectObject > 

< def Object ref="o2" /> 
</indirectObject> 
<directObject > 

<defObject ref="o4" /> 
25 </directObject> 

</sentence> 
</doSerial> 
<doParaIlel> 
30 <sentence> 

<subject > 

<defObject ref= M o2 M /> 
</subject> 
<vcrb> 

35 <facialExpressionVerb facialExpression="showHappiness" /> 

< /verb> 
</sentence> 
<sentence> 
40 < subject > 

<defObject ref="o3" /> 
</subject> 
<verb> 

<facialExpressionVerb facialExpression="showSadness ,r /> 

45 

</verb> 
</sentence> 

</doParallel> 

50 

</script> 
< / story > 



55 
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5 TABLE 5 

# 

#Walking Routine 
# 

10 def Rstep(person): 

anim = Do In Order { 
DoTogether ( 
#move leftarm forward 

person.larm.turn(backward, 1/8, Duration=0. 1), 



15 



20 



#rleg step forward 

person, dress, r thigh, turn (backward, 1/8, Duration=0. 1), 
person.dress.rthigh.rleg.turn(forward, 1/16, Duration=0. 1) 

), 



# person move and Ileg lift 
DoTogether ( 

person.dress.lthigh.lleg.turn(forward, 1/8, Duration=0.1), 
person, move (forward, 1/4, Duration=0. 1) 
25 ), 

#rleg come back to position 

person.dress.rthigh.turn(forward, 1/8, Duration=0. 1), 
person.dress.rthigh.rleg.turn(backward, 1/16, Duration=0. 1), 

30 

DoTogether ( 
#lleg come back to position 

person. dress.lthigh.Ueg.tum(backward, 1/8, Duration=0.1), 

35 #move leftarm back to position 

person. larm.turn(forward, 1/8, Duration=0. 1) 

), 
) 

40 return anim.Start() 

def Lstep (person): 

anim = DoInOrder ( 

45 DoTogether ( 

#move rightarm forward 

person.rarm.turn(backward, 1/16, Duration=0. 1), 

#lleg step forward 
50 person.dress.lthigh.turn(backward, 1/8, Duration=0.1), 

person, dress, lthigh.lleg. turn (forward, 1/16, Duration=0. 1) 

), 

#move and rleg lift 
55 DoTogether ( 

person.dress.rthigh.rleg.turn(forward, 1/8, Duration=0. 1), 
person.move(forward, 1/4, Duration=0. 1) 
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5 

#lleg come back to position 

person. dress. lthigh.turn(forwaxd, 1/8, Duration=0. 1), 
person, dress, lthigh.lleg. turn (backward, 1/16, Duration=0. 1), 

] 0 DoTogether ( 

#rleg come back to position 

person.dress.rthigh.rleg.turn(backward, 1/8, Duration=0. 1), 

#move rightarm back to position 
15 person.rarm.turn(forward, 1/ 16, Duration=0. 1) 

) 

return anim.Start() 

20 

def Walking(person): 

anim = DoInOrder ( 
Rstep(person), 
Lstep (person)) 
25 return anim. Start () 

def Walk (per son, steps): 

anim = loop (Walking (per son), steps) 
return anim.StartQ 

30 
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THE CLAIMS: 

1. A method for programming at least one synthetic creature (as defined 
herein) using a markup language to enable the synthetic creature to 
10 imitate the behaviour actions and expressions of humans and other living 

creatures, the method including the steps of: 

(a) creating a story, the story including at least one scene and at 
least one script; 

(b) the at least one scene including the at least one synthetic 
15 creature; 

(c) the script including at least one of a series of sentences; 

(d) each sentence including at least one subject and at least one 
action the subject is to perform. 

20 2. A method as claimed in claim 1, wherein the at least one action is 

determined by a verb selected from the group including: 
speech and speech intonation, facial expression, gestures, gaze, and 
motion. 

25 3. A method as claimed in claim 2, wherein speech and speech intonation 

include verbs that denote how the speech is delivered in order that the 
synthetic creature communicates with at least one other object. 

4. A method as claimed in claim 3, wherein the other object is another 
30 synthetic creature. 

5. A method as claimed in any one of claims 2 to 4, wherein speech 
intonation includes tonal variations in speech to give emphasis to at least 
one aspect of the speech and to express at least one emotion of the 

35 synthetic creature. 

6. A method as claimed in any one of claims 2 to 5, wherein facial 
expression includes verbs that denote various facial expression used to 
convey emotions. 



40 



7. A method as claimed in claim 6, wherein the emotion is selected from the 
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5 list including happiness, unhappiness, sadness, joy, surprise, anger, 

delight, tenderness, fear, loathing, love, pain, detached, enthusiastic, 
unfeeling, imitation, impassioned, and passion. 

8. A method as claimed in any one of claims 2 to 7, wherein gestures 
10 include verbs that denote gestures of body parts of the synthetic creature. 

9. A method as claimed in claim 8, wherein the body parts are selected from 
the list including head, shoulder, arms, forearms, hands, fingers, 
thumbs, chest, stomach, back, posterior, legs, ankles, feet, buttocks, 

15 eyebrows, nose, mouth. 

10. A method as claimed in any one of claims 2 to 9, wherein gaze includes 
verbs that denote movements of the eyes in order to look at another 
object and to thereby communicate with the other object. 

20 

11. A method as claimed in any one of claims 2 to 10, wherein motion 
includes verbs that denote movement from one location to another 
location. 



25 12. A method as claimed in claim 11, wherein the motion is selected from the 

list including approaching, retreating, entering, arriving, leaving, 
departing, arriving, crawling, walking, jogging, running, skipping, 
swimming, flying, falling, talking, trembling, tumbling, exercising, 
jumping, and rotating. 

30 

13. A method as claimed in any one of claims 2 to 12, wherein all verbs have 
an adverb speed property that denotes the speed with which the action is 
to be performed. 

35 14. A method as claimed in claim 13, wherein the speed is selected from the 

list including slow, normal, fast, moderate and accelerate. 

15. A method as claimed in any one of claims 1 to 14, wherein the sentence 
includes at least one direct object. 



40 



16. A method as claimed in any one of claims 1 to 15, wherein each sentence 
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5 includes at least one indirect object. 

17. A method as claimed in any one of claims 1 to 16, wherein each sentence 
includes at least one preposition phrase. 

10 18. A method as claimed in any one of claims 1 to 17, wherein the at least 

one scene also includes other objects with their relative positions. 

19. A method as claimed in any one of claims 1 to 18, wherein the program is 
implemented in XML. 

15 

20. A method as claimed in any one of claims 1 to 19, wherein the story is 
written in a user-selected language. 

21. A method as claimed in any one of claims 1 to 19, wherein after 
20 completion the program is parsed by a parser. 

22. A method as claimed in any one of claims 3 to 5, wherein the speech may 
be in a user-selected language. 

25 23. A graphic user interface to enable the method of any one of claims 1 to 22 

to be performed, the graphic user interface including: 

(a) a story display area; 

(b) a script preparation area; 

(c) a scene preparation area; and 
30 (d) a speech text area. 

24. A graphic user interface as claimed in claim 23, wherein the story display 
area enables the objects in a scene to be displayed to provide visual 
feedback to a user. 



35 



40 



25. A graphic user interface as claimed in claim 23 or claim 24, wherein the 
scene preparation area has a plurality of tools to enable an object to be 
included in the scene, for the object to be located in a required position, 
and for the object to be oriented in a required manner. 

26. A graphic user interface as claimed in claim 25, wherein the plurality of 
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tools include buttons and icons. 

27. A graphic user interface as claimed in any one of claims 23 to 26, 
wherein the script preparation area has a plurality of sub-areas for 
programming behaviour of a least one synthetic creature. 

28. A graphic user interface as claimed in claim 27, wherein the plurality of 
sub-areas include a program control box, an objects box, a verbs box, 
and a prepositions box. 

15 29. A graphic user interface as claimed in claim 27 or claim 28, wherein each 

of the plurality of sub-areas includes buttons and icons to enable a user 
to select an appropriate program item. 

30. A graphic user interface as claimed in claim 28 or claim 29, wherein each 
20 of the plurality of sub-areas has arrows to enable a user to browse 

available items. 

31. A graphic user interface as claimed in claim 28, wherein the program 
control box includes first and second areas to enable actions to be 

25 performed serially and in parallel respectively. 

32. A graphic user interface as claimed in claim 28 or claim 31, wherein the 
program control box includes a third area for creating sentences. 

30 33. A graphic user interface as claimed in claim 32, wherein when the third 

area is activated by a user, a template is displayed for the creation of a 
sentence. 

34. A graphic user interface as claimed in claim 33, wherein the template 
35 includes regions for each of: subject, verb, indirect object, direct object, 

preposition and object tuples. 

35. A graphic user interface as claimed in claim 34, wherein each preposition 
and object tuples represents a preposition phrase. 



40 



36. A graphic user interface as claimed in any one of claims 27 to 35, 
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5 wherein upon any one or more of the plurality of sub-areas being 

activated it causes an appropriate display in a program area. 

37. A computer markup language for the programming of a synthetic 
creature (as defined herein), the language including: 

10 (a) at least one object which is to perform at least one action, the 

object and the action arranged as a sentence; 

(b) a plurality of sentences being arranged as a script; 

(c) a scene corresponding to a script and including the at least one 
synthetic creature; 

15 (d) the scenes and scripts together creating a story. 

38. A language as claimed in claim 37, wherein the at least one object 
includes the synthetic creature. 

20 39. A language as claimed in claim 37 or claim 38, wherein the at least one 

action in defined by a corresponding verb. 

40. A language as claim in claim 39, wherein the at least one a verb is 
selected from the group including: speech and speech intonation, facial 

25 expression, gestures, gaze, and motion. 

41. A language as claimed in claim 40, wherein speech and speech 
intonation include verbs that denote how the speech is delivered in order 
that the synthetic creature communicates with at least one other object. 

30 

42. A language as claimed in claim 41, wherein the other object is another 
synthetic creature. 

43. A language as claimed in any one of claims 40 to 42, wherein speech 
35 intonation includes tonal variations in speech to give emphasis to at least 

on aspect of the speech and to express at least one emotion of the 
synthetic creature. 

44. A language as claimed in any one of claims 40 to 43, wherein facial 
40 expression includes verbs that denote various facial expression used to 

convey emotions. 
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A language as claim in claim 44, wherein the emotion is selected from the 
list including happiness, unhappiness, sadness, joy, surprise, anger, 
delight, tenderness, fear, loathing, love, pain, detached, enthusiastic, 
unfeeling, imitation, impassioned, and passion. 

A language as claimed in any one of claims 40 to 45, wherein gestures 
include verbs that denote gestures of body parts of the synthetic creature. 

A language as claimed in claim 46, wherein the body parts are selected 
from the list including head, shoulder, arms, forearms, hands, fingers, 
thumbs, chest, stomach, back, posterior, legs, ankles, feet, buttocks, 
eyebrows, nose, mouth. 

A language as claimed in any one of claims 40 to 47, wherein gaze 
includes verbs that denote movements of the eyes in order to look at 
another object and to thereby communicate with the other object. 

A language as claimed in any one of claims 40 to 48, wherein motion 
includes verbs that denote movement from one location to another 
location. 

A language as claimed in claim 49, wherein the motion is selected from 
the list including approaching, retreating, entering, arriving, leaving, 
departing, crawling, walking, jogging, running, skipping, swimming, 
flying, falling, talking, trembling, tumbling, exercising, jumping, and 
rotating. 

A language as claimed in any one of claims 40 to 51, wherein all verbs 
have an adverb speed property that denotes the speed with which the 
action is to be performed. 

A language as claimed in claim 51, wherein the speed is selected from the 
list including slow, normal, fast, moderate and accelerate. 

A language as claimed in any one of claims 37 to 52, wherein the 
sentence includes at least one direct object. 
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54. A language as claimed in any one of claims 37 to 53, wherein the 
sentence includes at least one indirect object. 

55. A language as claimed in any one of claims 37 to 54, wherein the 
10 sentence includes at least one preposition phrase. 

56. A language as claimed in any one of claims 37 to 55, wherein the 
language is implemented in XML. 

15 57. A language as claimed in any one of claims 37 to 56, wherein the scene 

includes other objects and their relative positions. 

58. A language as claimed in any one of claims 37 to 57, wherein the story is 
written in a user-selected language. 

20 

59. A language as claimed in any one of claim 41 to 43, wherein the speech 
may be a user- selected language. 

60. A language as claimed in any one of claims 35 to 55, wherein the 
25 language includes one or more selected from the list including text, 

images, audio, video, and other media formats. 

61. A programmable storage device readable by a machine, including a 
program of executable instructions to perform the method of any one or 

30 more of claims 1 to 22. 

62. A programmable storage device readable by a machine, including a 
program of executable instructions to display on a display screen a 
graphical user interface in accordance with any one or more of claims 23 

35 to 36. 

63. A programmable storage device readable by a machine, including a 
program of executable instructions in the language of any one or more 
claims 37 to 60. 



64. A graphical user interface in accordance with any one of claims 23 to 36 
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when used to perform the method of any one of claims 1 to 22. 



65. A method as claimed in any one of claims 1 to 22 using the language of 
any one of claims 37 to 60. 

10 66. A language as claimed in any one of claims 36 to 60 to perform the 

method of any one of claims 1 to 22 on a graphical user interface of any 
one of claims 23 to 36. 



67. A programmable storage device readable by a machine, including a 
15 program of executable instructions in the language of any one or more of 

claims 37 to 60 to perform the method of any one or more claims 1 to 22 
on a graphical user interface of any one or more of claims 23 to 36. 
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